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Abstract This paper describes a decision support system for military operations based on intelligent agents 
dedicated to the planning and shaping activities. The system has a 3D visualization toolkit for the modeling of 
the war virtual scenario, which is represented by a 3D mesh using a height map of the actual battle space. The 
intelligent agents approach is used to support the commanders' decisions in conformity with a known combat 
doctrine. Each troop is designed as an agent, where its actions are described as a set of prefixed action-rules. 



1 Introduction 

When fighting a battle, commanders must analyze and un- 
derstand both current and future combat scenarios in or- 
der to make strategic decisions, and also, plan and evaluate 
the force movements. Those activities are usually accom- 
plished with paper maps of the battle area, placed under 
sheets of acetate. This paper focuses the battlefield visuali- 
zation problem. 

2 Terrain Modeling 

One of the greatest challenges of the battiefield visualiza- 
tion problem is related to the acquisition cartographic data, 
and displaying it [1]. Height maps are used to create the 
digital terrain model. A regular grid based algorithm di- 
vides a large terrain into smaller patches using a recursive 
quadrantal routine. The well-known technique described 
by Rdttger [3] was improved in order to reduce the number 
of triangles. Rottger would divided the triangulated height 
field into a balanced quadtree to avoid cracks. The imple- 
mented method just divides the two tree nodes, which have 
a common edge with the more refined node. The remain- 
ing one is not divided. This approach reduces 8.738% the 
number of rendering triangles of Rottger method. Finally, 
a raster image of the actual battle area is mapped over the 
modeled surface applying texture procedures. This guaran- 
tees a virtual reality model where all the paper map infor- 
mation is available, like the existing mountains, rivers and 
roads around the battle area [2]. 

3 Troops Symbology 

The criteria for representing the troops over the virtual bat- 
tie space were based on the cartographic 2D simbology 
manual of the Brazilian Army. 3D objects were needed 
in order to be visible from oblique angles. For this rea- 
son the 2D ones were extruded into cubes, pyramids and 
spheres. The 3D unity objects might also have some pieces 
of representation, composed by characters, surrounding it. 
These characters can also become occluded. So, whenever 



the commander navigates through the scenario, all the text 
surrounding the 3D object rotates in order to keep facing 
the commander viewpoint. 

4 Intelligent Agents 

The agent engine is supplied by data extracted from the in- 
telligence reports received from the field. These reports are 
related to the seven interconnected knowledge groups. At 
first, agents can calculate its ability to engage on combat, 
during the day and night maneuvers. This is an absolute 
value, which means that it does not take into considera- 
tion the neighboring unities. The pool of agents evaluates 
the battlefield situation according to a knowledge base, and 
the troops receive a score for each of its knowledge groups. 
Then a report to a second kind of entity is made, which is 
responsible to consolidate the data and compute the rela- 
tive power of combat. This value is a quantitative number, 
which strict describes the relationship between the enemy 
fighting ability and allied one. 

5 Conclusion 

The implemented system has proved to be efficient when 
applied over the training of military troops. The mesh sim- 
plification is essential to allow the use of the application 
under poor graphical stations .The use of intelligent agents 
offers a tool for both the analysis and the efficiency evalua- 
tion of troops actions. 
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ABSTRACT 

The ability of future combat airlifters to deliver 
cargo into short, austere opportune landing sites which are 
very close to operating field units is expected to greatly 
improve the effectiveness of the US warfighter. But how 
do we find and select those sites in a quickly-changing 
battle scenario? The use of spectral image data from > 
satellites such as Earth Watch as the medium for 
identifying opportune landing sites is the concept of 
interest and is described herein. 

Ball Aerospace & Technologies Corporation 
(BATC) performed a study to explore the feasibility and 
data requirements for this concept. The study's objectives 
were two-fold: 

• Evaluate and demonstrate airlifter value; and 

• Evaluate and demonstrate opportune landing site 
selection concept feasibility. 

Spectral data was collected in several image 
resolutions for selected areas within the US. Potential 
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opportune landing sites were selected across large areas 
and the selection was refined based on higher resolution 
spectral images. Site surveys were used to grade the . 
accuracy of selection. The number of opportune landing 
sites (as short as 600 feet) relative to traditional runway 
lengths was determined and compared. 

BACKGROUND 

This paper summarizes Ball Aerospace & 
Technologies Corporation's (BATC) work effort under 
contract 6XY004217-9 in support of McDonnell Douglas 
Aerospace Corporation (MDC), Long Beach, CA, in their 
development of advanced technology transport aircraft. 
Other studies focusing on the use of remote sensing to 
identify opportune landing sites could not be identified. 

DATA AND DATA PROCESSING REQUIREMENTS 

An opportune landing site must support the 
performance characteristics, the performance limitations, 
and the associated takeoff/landing profiles of the aircraft 
using the landing site. The aircraft capabilities and 
limitations translate into attributes that describe the 
required physical characteristics of the ground and . 
surrounding area adjacent to the opportune site. These 
physical characteristics in conjunction with area themes, 
were used during the opportune site selection process to 
logically select areas that could support a useful runway. 
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Table 1. Aircraft Landing Requirements 



A irrra Tt nf Interest 

fill VI Mil VI lilltl Wl 


Minimum 
Landing 
Surface 

Length (ft) 


Minimum 
Landing 
Surface 

Width (ft) 


Minimum 
Runway 
Width (ft) 


Minimum CBR 
for 40 Passes 


C-130 (Line & Special Pilot) 


3500 • 


60 


130 


4.1 


C-130 (Special Pilot) 


2,500 


60 


130 


4.1 


SSTOL (Line Pilot) 


600 


.40. ■ 


130 


4.5 



Note: A CtiRof 4.1 Supports iah'^ing on high plasticity clay^tUff and hardersurfaces 



Table 1 shows the aircraft landing requirements for 
three aircraft/pilot variations that include a Supeir Short i 
Take-Off and Landing (SSTOL) Tiltwing aircraft flown by 
an inexperienced pilot (line Pilot), a C-130 aircraft flown 
by an experienced pilot (Special Pilot), and a C-130 
aircraft flown by an inexperienced pilot (Line Pilot). TheS : ■ 
requirements apply to sea-level, standard day for at 3.0G 
aircraft load configuration. 

The California Bearing Ratio (CBR) measures the 
hardness of an unprepared earth bearing surface. CBR 
is a function of the surface and subsurface mineral, u , . , y 
content, water content, and soil compression properties 
as well as the number of aircraft landings/takeoffs (passes) 
that it will support. 

The minimum runway width includes a flat portion 
on either side of a runway surface and an obstacle 
clearance slope on either side of the flat width. The 
minimum runway width allows for lateral movement pf the 
aircraft prior to touchdown and the presence of vertical ; : , 
obstructions (crops, bushes, signs, buildings; etc.) on cither 
side of the runway: •• \- •/ _>.yv. 

; BATC estijnated the minimum GBR for 40 passes ;: 
using derived CBRs for lrpass (provided by MDC . , ( - 
engineers) and a relationship derived from a formula 
presented in a Government report, [1]. The^derived ^- ; ; 
relationship. is: 

CBRx = [l + (x/100)(Coyerage4] ^ CBRr i; 

where CBRx is the CBR for Xnumber : qf passes and . 
coverage refers to the- number of times every point in the 
area is passed over by an^aircraft*wheeL Coverage is a 
function of the number of wheels ajvd the wheel sizes/ , 

Other considerations include the maximum slope of 
the landing surface and the maximum ground r,oughnessv :: , . 
The maximum slope of the landing surface applies to the 
incline/decline of the landing/surface: in the direction of 
the aircraft.; A 5° slope was assumed fox this analysis. : T ; i; . 
Maximum; ground roughness ;was not considered .during : : . t 
the selection of oppprtuije landing sites due to thev limited ( 
resolution. of available data. ■ : : . 

AreasJYithin the US Selected for Evaluation; 

... The initial goal of the feasibility study was to select 
areas within the cont^ental ILJS^that were most like four, 
international, regions, of interest Jliis strategy was : altered , 



due to the scarcity of affordable multispectral data. The 
\ f fihal;axeas ; of interest were chosen to make the most out of 
what was available. They include three 100-square mile 
scenes: The Black Hills near Spearfish, SD; a portion of 
Los Angeles, CA north of Palos Verdes; and an area north 
bfSalinas, CA. 

Spectral Image Data Requirements 

The type and number of categories that can be 
. -extracted from an image or map depend on the amount of 
,. v ^detail .the, map creator has available (and uses) to describe 
the location's features. Higher resolution data makes it 
easier to identify landing obstacles. Using 10-meter 
resolution data, the analyst is able to identify a single large 
tree and can identify the existence of buildings and houses. 
Using 100-meter data, the analyst can only identify a forest 
or large area of trees or flag cells as commercial or 
industrial areas. The 1-meter data would allow a more 
direct identification of obstacles to landing, since many 
obstacles causirig extensive damage to an aircraft could 
easily approachvl-meter dimensions. , v 

RATC. used multispectral data of various 
resolutions in a, progression to perform the analysis in each 
area of interest. The data in the progression included : . , 
100-meter seven-band Landsat ™ satellite data and 1- to : , 
3-meter four-band data collected bya conunercially flown .- 
airborne sensor system. The bands coyesec) the visible red, . 
blue, green range; the near-infrared; short-wave infrared, . .. . 

and long wave infrared. . ...... ..... . 

..... -Multi-band composite images ,wer;e used, to create.a 

fcrue.color, image composed of the blue, green, ai$ red , f 
bands or a false-color image composed of a variety of 
bands (visual or non-yisual) for determining areas . , 
populated with vegetation or for determining water and 
road boundaries. 

Features or themes were extracted from images to 
either acceptor reject a pixelas a suitable landing f 
attribute, "file themes included seven vegetation, thfee 
hydrography, seventeen geology, five land usage, and two 
man-made obstacle classifications. A truth table was 
charted using true or false notation to indicate whether the 
classified -pixel supported or precluded landings, : 
respectively. The assignment of T qr-Rfgr,;^ 
was dependent on ground moisture content. In these cases; 
the condition, wet or dry, was noted. ;For,ex i axnple i a bare , 
soil field will support landing during, dry; conditions butwill 
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Fig. L Site Selection Analysis Process 

not support landings when the soil turns to mud. The 
weather dependencies could be considered by the mission 
planners as one of the last factors for selecting a suitable 
landing area just hours prior to the planned sortie. 

SELECTION OF OPPORTUNE LANDING SUES 

Opportune landing areas were selected across 
large areas for several runway length variations and 
the selection was refined based on higher resolution 
spectral images. 

Figure 1 outlines the site selection analysis process. 
Several input variations were chosen to support the 
evaluation of data resolution requirements, the evaluation 
of SSTOL Tiltwing (Line Pilot) concept benefits over 
other airlifter aircraft, and the feasibility of the opportune 
site selection concept. The image data shown as an input 
to the overall process includes the 100-meter, 3-meter and 
1-meter resolutions for each of the three areas of interest. 
Data preparation was performed on images for each of the 
three data resolutions. Site selection logic was developed 
to automatically process each of the three large 100-square 
mile 100-meter resolution data images in order to create 
the initial landing regions of interest. 

BATC determined the number of landing site areas 
within each of the 100-square mile areas of interest (the 
Black Hills, Los Angeles, and Salinas) for three runway 
length variations. The runway variations of 600 feet, 2500 
feet, and 3500 feet were chosen to represent the typical 
runway requirements for SSTOL Tiltwing (Line Pilot), 
C-130 (Special Pilot), and C-130 (Line Pilot) cargo 
aircraft, respectively. BATC used the Potential Landing 
Site maps and the following criteria to count the number 
of landing site areas: 



At least 2 contiguous blocks (one by at least two 
landable pixels) were required to make up a 600-ft 
runway length and a 130-foot runway width 
landing area; 

• At least 8 contiguous blocks (one by at least eight 
landable pixels) were required to make up a 
2500-foot runway length and a 130-foot runway 
width landing area; 

• At least 1 1 contiguous blocks (one by at least 1 1 
landable pixels) were required to make up a 
3500-foot runway length and a 130-foot runway 
width landing area. 

BATC used 1- and 3-meter images to refine the site 
selection process. A sample of sites, chosen based on 
100-meter data resolution images, was revisited using 
features that were extracted from the higher resolution 
images. Obstacles, both man-made and natural, were the 
primary focus. An opportune site selected by the 
100-meter data was declared acceptable if no 1- or 3-meter 
pixels within the landing area contained non-landable 
criteria; acceptable with site preparation if 1 to 20 1-meter 
pixels within the landing area contained non-landable 
criteria; and non-acceptable if any 3-meter or more than 
20 1-meter pixels contained non-landable criteria. Figure 2 
illustrates the refinement process. 

SUE EVALUATION 

Site surveys were used to grade the accuracy of 
selection for a small sample of sites. Eight opportune 





Fig. 2. Site Selection Refinement 

landing areas in the Black Hills, and several others in the 
California areas of interest were surveyed to evaluate and 
rate the goodness of selection. The evaluation of selected 
opportune landing sites waa primarily qualitative and 
did not utilize.a qualified surveyor. Four tools and a 
digital camera were used to assist the site evaluation 
team. The tools included a hand-held GPS receiver, 
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a 200-foot ruler used for measuring ground distances, and 
a compass. The camera was used to record a visual 
image of the areas surveyed. 

An evaluation team visited each of the pre-selected 
surveyed landing areas and characterized the weather 
conditions, ground moisture content, and the growing 
season based on visual observation. After measuring the 
center coordinates of the landing site using a GPS 
receiver, the team rated the goodness of selection based 
on eight attributes: location, which was measured using the 
GPS receiver; size, which was measured using either a 
visual observation or the 200-foot ruler if the size was 
questionable; slope; ground hardness; the presence of 
man-made obstruction; and the presence of three classes 
of natural obstructions, vegetation, earth or rock, and ruts, 
creeks or swells. Using pre-defined criteria, the team rated 
and documented each attribute as either acceptable, 
acceptable with some site preparation, or unacceptable. 

The on-site survey of the eight sites in the Black 
Hills region was conducted on the 6th and 7th of 
November, 1996. The weather conditions were dry but 
cold. The ground was generally snow covered with the 
deepest snow in low-lying areas. The snow made the 
ground assessment more difficult, since the pre-survey 
analysis used non-snow covered images where rocks, ruts 
and swells were clearly visible. However, some areas of the 
ground were uncovered enough to recognize the general 
features which consisted of pasture grass with a scattering 
of flat and rough rocks ranging in size, with the largest 
several feet across. The area was populated by cattle farms 
and contained many small dry creeks and fence lines. 
Heavy tree areas were aligned along creek banks. 

The survey team eliminated four out of the eight 
opportune landing areas in the Black Hills identified by 
the 100-meter images. These four opportune landing areas 
were assessed to be unacceptable under all conditions. 
That is, site preparation would not be practical. Two 
additional sites of the remaining four were considered 
acceptable following site preparation. 

CONCEPT FEASIBILITY 

Following the site visit, BATC analyzed the results 
to assess the probability of correct selection and the 
"goodness-of-selection factor" for each of three pixel 
resolutions; 1-meter, 3-meter, and 100-meter, 

BATC grouped sites into one of three categories: 
useful landing sites with and without preparation; useful 
landing sites without site preparation; and non-useful 
landing sites. The grouping was based on the rating of the 
man-made and natural obstructions where acceptable, 
marginal and un-acceptable equated to: useful landing 
sites with and without preparation; useful landing sites 
without site preparation; and non-useful landing sites, 
respectively. The number of sites identified by each data 
resolution case and the resultant probability of correct 
selection is shown in Table 2. 



COMPARISON OF AIRCRAFT ALTERNATIVES — 
STATISTICAL RESULTS 

BATC considered three statistical metrics to 
compare differences between the variations in runway 



Table 2. Probability of Correct Selection 



CATEGORY OF 
USEFULNESS 


100- 
M 


3-M 


1-M 


SITE 
SURVEY 


Number of useful 
landing sites with or 
without preparation 


8 


4 


4 


4 


Probability of 
Correct Selection 


.5 


1 


1 


NA 


Number of useful 
landing sites not 
requiring site 
preparation 


8 


4 


2 


2 


Probability of 
Correct Selection 


.25 


.5 


1 


NA 



length. These metrics included: the number of opportune 
landing sites per Alpha area [a five square mile area 
adjacent to the Forward Line Own Troops (FLOT)]; the 
average distance from the FLOT to the closest landing 
area; and the percent of Alpha areas that are occupied by 
at least one opportune landing area. 

BATC used 100-meter, 3-meter, and 1-meter 
multispectral data to determine the number of opportune 
landing areas in each of the three areas of interest for each 
of three different runway length requirements: 600 feet, 
2500 feet and 3500 feet. The data is presented in terms of 
the number of landing sites per five square mile Alpha 
area. Since 20 Alpha areas are contained in each 
100-square mile area of interest, the number of opportune 
landing areas in each area of interest can be determined by 
multiplying the number of sites per five square mile area 
times 20. Results are shown for three conditions: 
non-degraded; site preparation could be required; and site 
preparation is not required. The number of sites found 
within each condition was determined by degrading the 
number of sites found by the 100-meter data. The first 
condition represents the 100-meter data with a 
degradation of 1. The second and third conditions 
represent the 100-meter data with a degradation of 0.5 and 
0.25 applied to the quantity of landing areas contained in 
each of the 20 Alpha areas. The number of sites for the 
"site preparation required condition" and the "no site 
preparation required condition" is consistent with the 
results of the 3-meter and 1-meter data, respectively. 
Sample results are presented graphically in Figure 3 for 
the Black Hills. A significant advantage is shown using 600 
foot runways. 

A comparison of the average distance from the 
FLOT to the closest landing area is presented graphically 
in Figure 4 for the Salinas area, which had the most 



46 



IEEEAES Systems Magazine, April !998 



Black Hills - Summary 
Opportune Sites 




■ 600 Ft Runway 

■ 2500 Fl Runwa r 
o 3500 Fl Runwa t 



Sita Prep 
Required 



No Site Prep 
Required 



Note: !*• 2500 tt end 3600 ft "no site preparation" site ts an airport 
with a 4200 ft runway located near Spaarfish in Alpha Area #2. 

Fig. 3. Number of Opportune Landing Sites 
Per Aircraft Type, The Black Hills 

dramatic differences among the three areas of interest. 
The differences noted in the Salinas area are due to the 
high availability of 600 foot landing areas for all FLOT 
positions. 

FINDINGS AND FUTURE INVESTIGATIONS 

The concept of identifying and locating forward 
opportune landing areas based on quality spectral data is 
feasible. The selection should involve a step-wise process 




• 600 Ft Runway 
■ 2500 Ft Runwa; 
D 3500 Ft Runwa 1 



Non Site Prep No Site 

Degraded Required Prep 

Fig. 4, Comparison of Average Distance from FLOT 
to the Closest Landing Area, Salinas, CA 



acceptable level. The use of level 1 Digital Terrain 
Elevation Data (DTED), which is the only DTED data 
available outside military intelligence channels, proved 
inadequate for discerning required slope variations. A 
finer slope measurement capability should be available 
from either 3- or 1 -meter stereo pair images. 

Ground hardness measurements (California 
Bearing Ratio) are limited by the spectral content of 
multispectral data. As a result, absolute values of CBRs 
were not determined in this study, although it was possible 
to provide discrimination over a range of CBR values. 
Hyperspectral data is expected to provide significant 
improvements to image texture analyses used for 
discerning object types and CBR values. Although this 
data is expected to be commercially available and 
affordable within the next several years, considerable data 
exists in the military intelligence community and could be 
made available to further expand on this application. 

The number of "600-ft" opportune landing sites 
identified in all three geographical areas analyzed 
exceeded the number of conventional landing sites (2500- 
and 3500-foot) by more than a factor of twenty. However, 
a small percentage of those sites selected could be visited 
and evaluated, mainly because of economic reasons. More 
work is needed to further refine the process and to 
increase the sample size through more on-site visitations. 

The findings of this study offer opportunities for 
significantly improving warfighting effectiveness and 
complement ongoing MDC efforts to show: 1) the timely 
airlift of key cargo items directly to a highly-mobile 
warfighting via opportune landing sites does impact battle 
outcome; and 2) combat airlifters can be designed and 
built to operate satisfactorily in and out of such landing 
areas. A parametric analysis of aircraft characteristics is 
continuing to determine optimal aircraft take-off and 
landing requirements. 



beginning with the application of low-resolution spectral 
data as a first sort on a large area of interest and should 
gradually apply higher-resolution data to refine the initial 
estimate. 

The 100-meter multispectral data is adequate to 
identify potential landing areas as a first sort. However, 
the 3-meter or 1-meter multispectral data is required to 
improve the probability of correct detection to an 
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Words from the Internet Not Found in Our Dictionaries - Yet !!! 

Neonphancy (ne on' fan see) n. A fluorescent light bulb struggling to come to life. 
Phonesia (fo nee' zhuh) n. The affliction of dialing a phone number and forgetting who you 

were calling just as they answer. 
Telecrastination (tel e kras tin ay' shun) n. The act of always letting the phone ring at least twice 

before you pick it up, even when you are only six inches away. 

(Contributions welcomed - send to Administrative Editor.) 



IEEEAES Systems Magazine, April J 998 



47 



search Abstract 



Page 1 of 2 



mm H#fe&£ i SEARCH * shop ( wss account i contact sees 



&*frtb*i'5.&fp PtibHotUfffts/Safvk.** Start rift ."dji Ca*f«fertf:*s Career*/ J6b$ 



Viral corns 

Us*it<S!rf States Patent sand Tfratottark Offices 



Help FAQ Terms IEEE Peer Review j Qui Ck Ll n ks 



» ABi 



j 

0" Stesctenfe 



O" W«! 



O&ftattnahiES 



0-&ceess1ft8 
IEEE fcfc 



Print Format 



Saarf.h Results [PDF FULL-TEXT 132 KB! EB£sL UEXL PQWNIQAP CITATION 



User-centered design and evaluation of a real-time 
battlefield visualization virtual environment 

Hix, D. Swan. J.E.. II Gabbard. J.L. McGee. M. Durbin, JL King. T. 

Virginia Polytech. Inst. & State Univ., Blacksburg, VA, USA; 

This paper appears in: Virtual Reality, 1999. Proceedings., IEEE 

Meeting Date: 03/13/1999 - 03/17/1999 

Publication Date: 13-17 March 1999 

Location: Houston, TX USA 

On page(s): 96 - 103 

Reference Cited: 17 

Number of Pages: xviii+299 

Inspec Accession Number: 6233294 

Abstract: 

The ever-increasing power of computers and hardware rendering systems has 
primarily motivated the creation of visually rich and perceptually realistic virti 
environment (VE) applications. Comparatively very little effort has been expe 
user interaction components of VEs. As a result, VE user interfaces are often | 
designed and are rarely evaluated with users. Although usability engineering 
emerging facet of VE development, user-centered design and usability evalua 
as a practice still lags far behind what is needed. This paper presents a struct 
iterative approach for the user-centered design and evaluation of VE user inte 
This approach consists of the iterative use of expert heuristic evaluation, folio 
formative usability evaluation, followed by summative evaluation. We describ- 
application of this approach to a real-world VE for battlefield visualization, de< 
resulting series of design iterations, and present evidence that this approach 
cost-effective strategy for assessing and iteratively improving user interactior 
VEs. This paper is among the first to report applying an iterative, structured, 
centered design and evaluation approach to VE user interaction design 

Index Terms: 

data visualisation human factors military computing real-time systems user centred 
interfaces virtual reality cost-effective expert heuristic evaluation real-time battlefiek 
visualization realistic virtual environment rendering summative evaluation usability < 
user interaction user interfaces user-centered design 



eee 



e eee g e ch ch b c 



be 



be 



•search Abstract 



Page 2 of 2 



Documents that cite this document 

Select link to view other documents in the database that cite this one. 



S&aE£hJBs.autts ffiJKiUkkTmJLSiKBl PREV NEXT POWNUOAP crTATKQN 



Ho;r ; e | Loq-out | Journals | Conference Proceedings | Standards | Search by A&tlPI | Basic Search j Mv.anced Se„arch | Join IEEE | Web Account | 
" New thi's week" j OPAC Linking Information | Your Feedback 1 Tech'nic'a't'Support I Email Alerting | No Robots Please | Release Notes | IEEE Online 

Publications I Help I FAQ| Terms | Sack to Toi> 

Copyright © 2004 IEEE — All rights reserved 



h eee e eee g e ch ch b c 



be 



be 



User-Centered Design and Evaluation of a Real-Time Battlefield Visualization 

Virtual Environment 

Deborah Hix 1 , J. Edward Swan II 2 , Joseph L. Gabbard 3 , Mike McGee 1 , Jim Durbin 2 , Tony King 2 

Virginia Polytechnic Institute and State University, Blacksburg, VA 
2 The Naval Research Laboratory, Washington, D.C. 
Virtual Prototyping and Simulation Technologies, Inc., Blacksburg, VA 



Abstract 

The ever-increasing power of computers and hardware render- 
ing systems has, to date, primarily motivated the creation of 
visually rich and perceptually realistic virtual environment (VE) 
applications. Comparatively very little effort has been expended 
on the user interaction components of VEs. As a result, VE user 
interfaces are often poorly designed and are rarely evaluated 
with users. Although usability engineering is a newly emerging 
facet of VE development, user-centered design and usability 
evaluation in VEs as a practice still lags far behind what is 
needed. 

This paper presents a structured, iterative approach for the 
user-centered design and evaluation of VE user interaction. 
This approach consists of the iterative use of expert heuristic 
evaluation, followed by formative usability evaluation, followed 
by summative evaluation. We describe our application of this 
approach to a real-world VE for battlefield visualization, de- 
scribe the resulting series of design iterations, and present evi- 
dence that this approach provides a cost-effective strategy for 
assessing and iteratively improving user interaction design in 
VEs. This paper is among the first to report applying an itera- 
tive, structured, user-centered design and evaluation approach 
to VE user interaction design. 

Keywords: user-centered design, user interfaces, user interac- 
tion, user assessment, usability engineering, usability evaluation, 
virtual environments, virtual reality, expert heuristic evaluation, 
formative evaluation. 

1 Introduction and Related Work 

Despite the ever-increasing power of computers and hardware 
rendering systems, the user interaction components of VE appli- 
cations are often poorly designed and are rarely evaluated with 
users. The vast majority of VE research and design effort has 
been on the development of visual quality and rendering effi- 
ciency. As a result, many visually compelling VEs are difficult 
to use and are, therefore, non-productive for their users. While 
these VEs might make good entertainment applications, their 
usability problems prevent them from being useful for effi- 
cienUy solving real-world problems. 

Usability engineering [10] and user-centered design [11] are 
newly emerging facets of VE design and evaluation. VE de- 
signers and developers are becoming aware of traditional hu- 



man-computer interface (HCi) usability research and are begin- 
ning to apply and expand upon those methods for VEs. A few 
efforts have been reported to date; however, user-centered de- 
sign and usability evaluation in VEs as a practice still lags far 
behind what is needed. 

One reported work on user-based evaluation in VEs is 
Bowman et al. [1], who investigated an aspect of navigation in 
VEs and present a framework for evaluating travel (viewpoint 
motion control). The framework supports a methodology for 
evaluating different VE travel techniques and for appropriately 
matching travel techniques with virtual applications. Several 
aspects, or quality factors, were identified as being important to 
travel: speed, accuracy, spatial awareness, ease of learning, in- 
formation gathering, presence, and user comfort. The authors 
acknowledge that task-related factors (task, environment, user, 
and system characteristics) can have a greater impact on quality 
factor performance than the travel technique selected. The 
evaluation methodology described is intended to be generaliz- 
able to a variety of VEs. 

Salzman et al. [14] discuss how usability engineering meth- 
ods shaped iterative development of a VE designed for educat- 
ing students on various concepts associated with Newton's laws 
of physics. The goal of the design process was to develop a 
usable and educational virtual world. The authors applied us- 
ability evaluation to identify and refine early system weaknesses 
across three premises: usability, learning, and learning vs. us- 
ability. Both potential users (high school students) and experts 
in the field (physics professors) participated in the formative 
evaluations, which resulted in changes that improved the final 
VE user interaction. 

Other research that has reported a limited element of usabil- 
ity evaluation includes a study of hap tic interfaces [6], and an 
investigation of spatial input devices [7]. In addition, Stuart [16] 
describes basic methods for evaluating general usability compo- 
nents of VEs. 

While these efforts provide insights about usability issues of 
specific VE technology, most do not provide sufficient breadth 
for large, complex VE design and assessment. Gabbard and Hix 
[4] propose a framework of usability characteristics structured to 
support usability engineering of VEs. They present a methodol- 
ogy for approaching design and assessment of VE user inter- 
faces, which employs a top-down, step-wise refinement of VE 
usability space. This framework was used during evaluation of 
the battlefield visualization VE described herein (see Section 4.3 
and Section 5). 



Figure 1: Screen shot from the Dragon battlefield visualization virtual environment. 



Personnel at the Naval Research Laboratory's (NRL) Virtual 
Reality Lab have developed a VE for battlefield visualization, 
called Dragon (Figure 1) [3], which is implemented on a Re- 
sponsive Workbench [9, 13]. The responsive workbench pro- 
vides a natural metaphor for visualizing and interacting with 
three-dimensiona computer-generated scenes using a familiar 
tabletop environment. Applications in which several users col- 
laborate around a workspace, such as a table, are excellent can- 
didates for the workbench. Researchers from NRL, collabora- 
tively with researchers from Virginia Tech, are empirically 
studying the most important usability parameters of an effective 
VE user interface for Dragon. 

In the next section, we discuss battlefield visualization in 
general, and we describe the Dragon battlefield visualization 
VE. In Section 3, we discuss three important usability evalua- 
tion methods that can be profitably applied to VEs: expert heu- 
ristic evaluation, formative evaluation, and summative evalua- 
tion. In Section 4 we present our methodological approach for 
applying expert heuristic and formative evaluation methods to 
Dragon's design and evaluation, and in Section 5 we describe 
and discuss the design iterations that resulted from using this 
approach. In Section 6, we discuss lessons learned from this 
work, including evidence that our structured approach provides 



a cost-effective strategy for assessing and iteratively improving 
user interaction designs in VEs. We conclude with ideas for 
future work, particularly summative evaluation. 

2 The Dragon Real-Time Battlefield 
Visualization Virtual Environment 

2.1. Battlefield Visualization and Dragon 

For decades, battlefield visualization has been accomplished by 
placing paper maps of the battlespace under sheets of acetate. 
As intelligence reports arrive from the field, technicians use 
grease pencils to mark new information on the acetate. Com- 
manders then draw on the acetate to plan and direct various 
battlefield situations. Thus, the map and acetate together present 
a visualization of the battlespace. Using maps and overlays can 
take several hours to print, distribute, and update. Historically 
(before high-quality paper maps) these same operations were 
performed on a sandtable (a box filled with sand shaped to rep- 
licate the battlespace terrain). Commanders moved around 
small physical replicas of battlefield objects to direct battlefield 
situations. Currently, the fast-changing modem battlefield pro- 
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duces so much time-critical information that these cumbersome, 
time-consuming methods are inadequate for effectively visual- 
izing the battlespace. 

In Dragon, the workbench provides a three-dimensional dis- 
play for observing and managing battlespace information shared 
among commanders and other battle planners. Visualized in- 
formation includes a high-resolution terrain map; entities repre- 
senting friendly, enemy, unknown, and neutral units; and sym- 
bology representing other features such as obstructions or key 
battle objectives. Dragon receives electronic intelligence feeds 
that provide constantly updated, displayable information about 
each entity's status, including position, speed, heading, damage 
condition, and so forth. Users can navigate to observe the map 
and entities from any angle and orientation, and can query and 
manipulate entities. 

2.2. Design of User Interaction in Dragon 

Early in Dragon's development, we developed and assessed 
three general interaction methods for the workbench, any of 
which could have been used to interact with Dragon: hand ges- 
tures using a pinchglove [12], speech recognition, and a hand- 
held flightstick. Although an interesting possibility for VE in- 
teraction, we found speech recognition still too immature for 
battlefield visualization, and we found the pinchglove to be 
fragile, time-consuming to pass from user to user, and limiting 
in that it requires right-handed users whose hands are approxi- 
mately the same size. In contrast, we found the hand-held 
flightstick to be robust, easily handed from user to user, and 
applicable to both right- and left-handed users. 

Based on these observations, we modified a three-button 
game flightstick by removing its base and placing a six degree- 
of-freedom position sensor inside. We tracked the flightstick' s 
position and orientation relative to an emitter located on the 
front center of the workbench. We accomplished VE interaction 
with a virtual laser pointer metaphor: a laser beam appears to 
come out of the flightstick, allowing interaction with the terrain 
or object that the beam intersects. 

Early in its development, when very little usability evalua- 
tion had been performed, Dragon was demonstrated as a proto- 
type system at two different military exercises. In both demon- 
strations, an objective was a proof-of-concept for using a work- 
bench-based battlefield visualization tool. Feedback from both 
civilian and military VIPs indicated that users found Dragon's 
battlespace visualization to be more effective and efficient than 
the traditional method of maps, acetate, and grease pencils. 
Following these successful demonstrations, we began intensive 
usability evaluations and iterations of Dragon's user interface. 

3 Usability Evaluation Methods 

User-based evaluation is an essential component of developing 
any interactive application, and is especially important for appli- 
cations as complex and innovative as VEs. Three kinds of us- 
ability evaluation are particularly appropriate: expert heuristic 
evaluation, formative evaluation, and summative evaluation. 
We performed the first two types extensively during Dragon's 
development (Sections 4 and 5), and have plans for the third 
type (Section 6). 

Expert heuristic evaluation [10] is a type of analytical evalua- 
tion in which an expert in user interaction design assesses a 
particular user interface by determining what usability design 



Figure 2: Formative evaluation process. 

guidelines it violates and supports. Then, based on these find- 
ings, especially the violations, the expert makes recommenda- 
tions for changes to improve the design. In the case of VEs, this 
is particularly challenging because there are so few guidelines 
that are specific to VE user interfaces. Thus, users are not di- 
rectly involved in expert heuristic evaluation. Typically, this 
type of usability evaluation is more effective if the experts are 
not also developers of the user interaction design being evalu- 
ated. This was our situation: the first three authors of this paper, 
who were not involved with development of Dragon, did much 
of the expert heuristic evaluation described in Section 4.3. 

Formative evaluation [8] is a type of empirical, observational 
assessment with users that begins in the earliest phases of user 
interaction design and continues throughout the entire life cycle. 
Formative evaluation produces both qualitative (narrative) and 
quantitative (numeric) results. The purpose of formative eval- 
uation is to iteratively and quantifiably assess and improve the 
user interaction design. 

An important point to note in the formative evaluation proc- 
ess, shown in Figure 2, is that both qualitative and quantitative 
data are collected from representative users during their per- 
formance of task scenarios. Developers often have the false 
impression that usability evaluation is something rather warm 
and fuzzy, with no "real" process and collecting no "real" data. 
Quite the contrary is true; experienced usability evaluators col- 
lect large volumes of both qualitative data and quantitative data 

Qualitative data are typically in the form of critical incidents 
[5, 8]. A critical incident occurs while a user is performing task 
scenarios, and is an event that has a significant effect, either 
positive or negative, on user task performance or user satisfac- 



tion with the interface. Events that affect user performance or 
satisfaction therefore have an impact on usability. Typically, a 
critical incident is a problem that a user encounters (e.g., an 
error, being unable to complete a task scenario, confusion, etc.). 
Section 5 describes the major design iterations that resulted from 
hundreds of critical incidents, which we collected during our 
formative evaluation studies. 

Quantitative data are generally related, for example, to how 
long it takes and the number of errors committed while a user is 
performing task scenarios. These data are then compared to 
appropriate baseline metrics. Quantitative data generally indi- 
cate that a problem has occurred; qualitative data indicate where 
(and sometimes why) it occurred. 

Collection of both these types of data is an important part of 
the formative evaluation process. While we focused largely on 
qualitative, critical incident data, we also collected some quan- 
titative data. 

Summative evaluation [8], in contrast, is an empirical assess- 
ment with users of an interaction design in comparison with 
other interaction designs for performing the same user tasks. 
Summative evaluation is typically performed when there are 
some more-or-less < *final" versions of the interaction designs, 
and it yields primarily quantitative results. The purpose of 
summative evaluation is to statistically compare user perform- 
ance with different interaction designs, for example, to deter- 
mine which one is better, where "better" is defined in advance. 
Summative evaluations of Dragon are planned (Section 6). 

Best guesses about an interaction design are substantiated or 
refuted by many tight, short cycles of heuristic and formative 
evaluation. During the design and assessment of the Dragon VE 
user interface, we performed numerous cycles of heuristic and 
formative evaluation — some as short as a few minutes (these 
were the really bad designs!), others lasting several hours. 
Evolution of essentially all decisions about design details came 
from many rounds of evaluation. As discussed in the following 
sections, from the heuristic and formative evaluations we have 
greatly improved Dragon's user interaction design, and are now 
planning a summative study. 

4 Method: Application of Design and 
Evaluation Methods 

4.1 Focus on Navigation 

During our early demonstrations and evaluations, we observed 
that navigation — how users manipulate their viewpoint to 
move from place to place in a virtual world (in this case, the 
map for battlefield visualization) — profoundly affects all other 
user tasks. If a user cannot successfully navigate in a virtual 
world, then other user tasks (e.g., involving specific objects or 
groups of objects) simply cannot be performed. A user cannot 
query an object if the user cannot navigate through the virtual 
world to get to that object. Although we performed a user task 
analysis before our heuristic and formative studies, these studies 
corroborated our task analysis and our expectations of what 
tasks are most important. 

Further, our observational studies revealed several other ge- 
neric tasks performed by users of battlefield visualization VEs, 
including object manipulation, object selection, object querying, 
query response, and object aggregation. These user tasks will 
become the focus of possible future research for us and for oth- 



ers. Again, without having performed the expert and formative 
usability evaluations, we would only be able to guess at our 
assumptions about user tasks. 

4.2 Methodology 

We used the basic Dragon application as an instrumentable test- 
bed, modified as needed for our heuristic and formative usability 
evaluation purposes. We performed extensive evaluations over 
a nine-month period, using anywhere from one to three users for 
each cycle of evaluation. From a single evaluation session, we 
often uncovered design problems so serious that it was pointless 
to have a different user attempt to perform the scenarios with the 
same design. So we would iterate the design, based on our ob- 
servations, and begin a new cycle of evaluation. We went 
through four major cycles of iteration (Section 5). 

Based on our task analysis and early evaluations, we created 
a set of scenarios comprised of benchmark user tasks, carefully 
considered for coverage of specific issues related to navigation. 
For example, some of the tasks exploited an ego-centric (user 
moves through world) navigation metaphor while others ex- 
ploited an exo-centric (user moves the world) navigation meta- 
phor (see Section 5). Some scenarios exercised various naviga- 
tion tasks (i.e., degrees of freedom: pan, zoom, rotate, heading, 
pitch, roll) throughout the virtual map world. Other scenarios 
served as primed exploration or non-primed searches [2], while 
still others were designed to evaluate rate control versus position 
control in the virtual world. We thoroughly pre-tested and "de- 
bugged" all scenarios before presenting them to users during an 
evaluation session. 

43 Expert Heuristic Evaluations 

During our expert heuristic evaluations, various user interaction 
design experts worked alone or collectively to assess the evolv- 
ing user interaction design for Dragon. In our earliest heuristic 
evaluations, the experts did not follow specific user task sce- 
narios per se, but engaged simply in "free play" with the user 
interface. All experts knew enough about the purpose of Dragon 
as a battlefield visualization VE to explore the kinds of tasks 
that would be most important for users of Dragon. During each 
heuristic evaluation session, one person was typically "the 
driver," holding the flightstick and generally deciding what and 
how to explore in the application. One and sometimes two other 
experts were observing and commenting. Much discussion oc- 
curred during each session. 

As mentioned earlier, the first three authors of this paper 
were often the experts assessing the current design. Their as- 
sessment and discussions were guided largely by their own 
knowledge of interaction design for VEs, and, more formally, by 
a framework for usability characteristics of VEs [4], discussed in 
Section 1 . This framework provided a more structured means of 
evaluation than merely wandering around at random in the ap- 
plication, and provided guidance on how to make modifications 
to improve discovered design guideline violations. The major 
design problems uncovered by the expert heuristic evaluations 
were: 1) poor mapping of navigation tasks (e.g., pan, zoom, 
pitch, heading) to flightstick buttons, 2) missing functionality 
(e.g., exo-centric rotate, terrain following), 3) problems with 
damping of map movement in response to flightstick movement, 
and 4) graphical and textual feedback to the user about the cur- 
rent navigation task (e.g., pan, zoom, etc.). These problems, and 



how we addressed them, are discussed further in Section 5. 
After our cycles of expert heuristic evaluation had revealed and 
remedied as many design flaws as possible, we moved on to 
formative evaluations. 

4.4 Formative Evaluations 

During each of six formative evaluation sessions, we followed a 
formal protocol of welcoming the user, giving them an overview 
of the evaluation about to be performed, and then explaining the 
responsive workbench and the Dragon application. We were 
careful to not explain too many details of the Dragon interaction 
design, since that was what we were evaluating. Then the user 
was asked to play with the flightstick to figure out which button 
activated which navigation task (e.g., pan, zoom, etc.). We 
timed each user as they attempted to determine this, and took 
notes on comments they made and any critical incidents that 
occurred. Once a user had successfully figured out how to use 
the flightstick, we began having them perform the scenarios. If 
about 15 minutes passed without a user figuring out the flight- 
stick and its buttons (this happened in only one case), we filled 
in details that they had not yet determined and moved on to sce- 
narios. 

Time to perform the set of scenarios ranged from about 20 
minutes to more than an hour. We timed user performance of 
individual tasks and scenarios, and counted errors they made 
during task performance (quantitative data). A typical error was 
moving the flightstick in the wrong direction for the particular 
navigation metaphor (exo-centric or ego-centric) that was cur- 
rently in use. Other errors involved simply not being able to 
maneuver the map (e.g., to rotate it) and persistent problems 
with mapping navigation tasks to flightstick buttons. Again, 
these are discussed further in Section 5. We also carefully noted 
critical incidents, especially related to errors, and constructive 
comments users made about the design (qualitative data). 

During each session, we had at least two and often three 
evaluators present: one was the "leader" who ran the session 
and interacted with the user; the other one or two evaluators 
recorded timings, counted errors, and collected qualitative data. 
While both the expert heuristic evaluation sessions and the for- 
mative evaluation sessions were personnel-intensive (with two 
or three evaluators involved), we found that the quality and 
amount of data collected by multiple evaluators greatly out- 
weighed the cost of those evaluators. After each session, we 
analyzed both the quantitative and qualitative data, and based 
our next iteration on our results, as explained in the next section. 

5 Results: Iterations of the Dragon User 
Interaction Design 

Table 1 summarizes the four major iterations of the Dragon user 
interaction design over an approximately one-year period. It 
gives a high-level description of each iteration (including both 
visual and flightstick characteristics), and indicates the major 
usability findings for each iteration. (Space does not permit us 
to explain all the information in this table in detail.) Our find- 
ings, shown in rows of the table, fell into four categories: 

General Description. For each iteration, we give a brief de- 
scriptive title in the top four cells of Table 1. A general descrip- 
tion of each iteration's most salient features is shown beneath, 
along with the approximate date when the iteration was com- 
pleted. 



Interaction Description. This category describes some specif- 
ics of how a user interacts with each design iteration. We ex- 
perimented extensively with variants of two different navigation 
metaphors (described below): exo-centric and ego-centric. We 
visualized the virtual laser pointer (see Section 2.2) by drawing 
a beam corning out of the flightstick and intersecting the envi- 
ronment. In the first ("Virtual Sandtable") iteration, we also 
drew a skeletal hand "holding" the beam to visualize the user's 
hand (lower edge of Figure 1). This category of Table 1 also 
shows the degrees of freedom used by the flightstick tracker. 

Device Description. This category defines the mappings from 
the three flightstick buttons (left, right, and trigger) to degrees of 
freedom; examples are explained below. 

Evaluation Results. This category indicates which evaluations 
were performed on each iteration, and summarizes major 
strengths and major flaws of each. The last row of Table 1 
summarizes our user interaction design modification recommen- 
dations to Dragon's programmers. 

During early design, we implemented two navigation metaphors: 
exo-centric (or map-centric) and ego-centric (or user-centric). 
An exo-centric navigation metaphor is based on how a user 
would interact with a real physical map on a table. Different 
buttons are used for navigation tasks such as pan, zoom, and 
pitch. The map mimics the motion of the flightstick, so that the 
map acts as if it is stuck to the laser beam; user movement of the 
flightstick in any direction causes the map to move in that same 
direction. The magnitude of a user's gesture controls the dis- 
tance of the map's movement in the virtual world (this is also 
called zero-order motion). This means that, for example, when 
panning from side to side of a zoomed-in map, a user must make 
repeated panning gestures, each of which translates the map a 
distance equivalent to the length of the user's gesture. 

An ego-centric navigation metaphor is loosely based on the 
concept of a user flying above the map as if in an airplane. 
Various button combinations are again used for navigation tasks. 
The magnitude of a user's gesture controls the velocity of the 
map's movement (also called first-order motion); for example, a 
user can fly from one side to the other of a zoomed-in map with 
a single gesture. 

The first iteration, "Virtual Sandtable", was based on the 
sandtable concept briefly described in Section 2.1, and was the 
version demonstrated in the military exercises mentioned in 
Section 2.2. So in addition to expert heuristic evaluation, we 
had feedback from the demonstrations. A key finding of this 
iteration was that users wanted a terrain-following capability, 
allowing them to "fly" over the map. Based on observations of 
users interacting with maps in a combat center, we had initially 
thought that a battlespace visualization application only required 
an exo-centric navigation metaphor. In reality, the workbench- 
based Dragon creates a very rich environment, in which users 
can do much more than just move a map. They can actually 
experience the environment by visually sizing up terrain fea- 
tures, entity placement, fields of fire, lines of sight, and so forth. 
Exo-centric navigation worked well when globally manipulating 
the environment and conducting operations on large-scale units. 
However, for small-scale operations, users wanted the "fly" 
capability. The logical approach to designing this into Dragon 
was an ego-centric flying capability. We found that the map- 
ping of flightstick buttons to navigation tasks shown in Table 1 
(i.e., trigger and left button pressed simultaneously produced 
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combined pan and zoom; trigger and right button together pro- 
duced combined heading and pitch) worked well for users. 

In designing the second iteration, "Point and Go," we used 
the framework of usability characteristics of VEs [4] (see Sec- 
tion 1) to suggest various possibilities for an ego-centric naviga- 
tion metaphor design, such as W1M [15] and eye-in-hand [17]. 
We ultimately designed a '"point and go" metaphor, in which we 
attempted to avoid having different modes (and buttons) for 
different navigation tasks (pan, zoom, etc.) because of known 
usability problems with moded interaction. Further, we based 
this decision on how a person often navigates to an object or 
location in the real world; namely, they point (or look) and then 
go (move) there. Our reasoning was that adopting this same 
idea to ego-centric navigation would simplify the design and at 
least loosely mimic the real world. So in this iteration, a user 
simply pointed the flightstick toward a location or object of 
interest, and pressed the trigger to fly there. We found through 
our expert heuristic evaluation that the single gesture to move 
about was not powerful enough to support the diverse, compli- 
cated navigation tasks inherent in Dragon. Furthermore, a single 
gesture meant that all degrees of freedom were controlled by 
that single gesture. This resulted in, for example, unintentional 
rolling when a user only wanted to pan or zoom. Essentially, we 
observed a control versus convenience trade-off. Many naviga- 
tion tasks (modes) were active simultaneously, which was con- 
venient but difficult to physically control. With separate tasks 
(modes), there was less convenience but physical control was 
easier because degrees of freedom were more limited in each 
mode. In addition to these serious problems, we found that us- 
ers wanted to rotate around an object, such as to move com- 
pletely around a tank and observe it from all sides. This indi- 
cated that Dragon needed an exo-centric rotate ability, which 
was added. This interesting finding showed that neither a pure 
ego-centric nor a pure exo-centric metaphor was desirable; each 
metaphor has aspects that are more or less useful depending on 
user goals. 

In the third iteration, "Modal," we went from the extreme of 
all navigation tasks coupled on a single button to a rather oppo- 
site design in which each navigation task was a separate mode. 
Specifically, as a user clicked the left or right flightstick button, 
Dragon cycled successively through the tasks of pan, zoom, 
pitch, heading, and exo-centric rotate. Once a user had cycled to 
the desired task, it was enabled and thus accessible from the 
trigger, and the task name appeared in a small textual indicator. 
We observed that, as we expected, it was very cumbersome for 
users to always have to cycle between modes, and it was obvi- 
ous that we still had not achieved a compromise between con- 
venience and control. Again using the framework of usability 
characteristics of VEs [4] for guidance, for our fourth iteration 
of the Dragon interaction design, "Integrated Navigation," we 
decided to couple pan and zoom onto the flightstick trigger, 
pitch and heading onto a single button, and exo-centric rotate 
and zoom onto the third flightstick button, as indicated in Table 
1. Our fourth generation design appears to have achieved the 
desired convenience versus control compromise. In our final 
evaluation studies, we found that at last we had a design for 
navigation that seemed to work well for most users. The only 
problem we observed was minor: damping of map movement 
was too great and needed some adjustment, which we made. 



Figure 3: Types of usability evaluation and their cost. 
6 Lessons Learned and Future Work 

A key finding of our research is the successful progression from 
heuristic to formative to summative evaluations as a very cost- 
effective strategy for assessing and improving a user interaction 
design. Far too often, summative studies are conducted on ap- 
plications whose interaction design has had little or no heuristic 
or formative evaluation. This situation is unfortunate because it 
is often the case that very expensive summative evaluations are 
comparing "good apples" with "bad oranges". That is, the dif- 
ferences between two interaction designs may occur because one 
design is inherently better, in terms of usability, than the other. 
If both designs have been heuristically and/or formatively evalu- 
ated, then experimenters can have confidence that the interaction 
designs are essentially equivalent in terms of their usability: they 
will be comparing "good apples" to "good oranges". And it is 
therefore much more likely that any differences found in a 
summative comparison are truly due to differences in the nature 
of the applications, and not in their user interaction designs per 
se. 

Further, the cost of performing these three types of evalua- 
tions typically ranges from lowest for expert heuristic evalua- 
tions to highest for summative evaluations, as shown in Figure 
3. So if expert heuristic evaluations are not performed prior to 
formative evaluations, the formative evaluations will typically 
take longer and require more users, and yet reveal many of the 
same usability problems that could generally have been discov- 
ered by less expensive heuristic evaluations. Thus, expert heu- 
ristic evaluations can reduce the cost of formative studies, and 
formative studies produce interaction designs that are truly com- 
parable in summative studies for uncovering differences be- 
tween applications. 











Integrate NaVistfiQR 




sandtable metaphor 


one gesture moves anywhere 
on map 


all navigation tasks sepa- 
rated into discrete modes 


modes mapped to all three 
fliqhtstick buttons 


Approximate Date 


June 1997 


November 1997 


January 1998 


April 1998 . 





Navigation Metaphor 


exo-centric (map-centric) 


ego-centric (flying) 


primarily ego-centric, except 
for exo rotate 


primarily ego-centric, except 
for exo rotate 


Laser Pointer visual 
Representation 


laser pointer & skeleton hand 


laser pointer 


laser pointer 


laser pointer 


Supported Degrees of 
Freedom 


x, y, z, heading, pitch 


x, y, z, heading, pitch, roll 


x, y, z, heading, pitch 


x, y, z. heading, pitch 





Button Mappings 


trigger & left->pan & zoom 
trigger & right— sheading & 
pitch 


trigger— >pan & zoom & pitch 
& heading & roll 


left and right buttons cycle 
modes: pan, zoom, pitch, 
heading, exo rotate 


trigger— >pan & zoom 
left — >pitch & heading 
riqht-»exo rotate & zoom 








Evaluations 
Performed 


heuristic 


heuristic 


heuristic and formative 


heuristic and formative 


Major Strengths of 
Iteration 


• easy to pan/zoom 

• good for overview tasks 


• modeless navigation 


• easy navigation to any 
location with single mode 


• easy navigation to any 
location 

• easy to switch between 
naviqation tasks 


Major Flaws of 
Iteration 


• skeleton hand orientation 
did not match user hand 
orientation 

• terrain following difficult 

• pan gesture parallel to floor 
not workbench screen 


• hard to travel to non-visible 
location on map 

• could travel underneath 
map 

• trigger overloaded with too 
many degrees of freedom 

• many navigation tasks 
resulted in unintentional 
rollinq 


• too cumbersome to switch 
between modes 


• too much damping; user 
movement too slow 

• zoom gesture parallel to 
workbench screen, not floor 


Recommendations to 
Programmers for 
Interaction Design 
Changes 


• support terrain following 


• fine-tune damping and 
acceleration 

• add collision detection with 
map 

• remove ability to roll 

• add exo-centric rotation 


• couple modes so that only 
three navigation modes 
because then can map to 
three buttons on flightstick 

• couple pitch and heading 

• couple pan and zoom 


• fine-tune damping and 
acceleration 



Table 1: Major iterations of Dragon user interaction design. 



Our future work will focus on summatively evaluating our 
current navigation design. During our expert heuristic and for- 
mative evaluations, we discovered many different variables that 
affect navigation usability in VEs. We have narrowed this (ini- 
tially large) list to five variables, based on the framework of 
usability characteristics [4], our observations during heuristic 
and formative evaluations, and our expertise in VE interaction 
design. We feel these Ave variables have the greatest effect on 
navigation, and are therefore the most important candidates for 
summative evaluations: 

1) navigation metaphor (ego- vs. exo-centric), 

2) gesture control (controls rate vs. controls position), 

3) visual presentation device (workbench, desktop, CAVE 1 "), 

4) head tracking (present vs. not present), and 

5) stereopsis (present vs. not present). 



An expected result of these planned studies is empirically de- 
termined guidelines for navigation design in VEs. 

To summarize, our research has produced results at three levels: 

1) important navigation improvements in Dragon, 

2) recommendations for navigation design in VEs, especially 
workbench-based VEs, and 

3) evidential substantiation of a structured approach for user- 
centered design and evaluation of VEs. 

This paper is one of the first to report using expert heuristic 
evaluation followed by formative usability evaluation as a 
structured approach to the iterative, user-centered design and 
evaluation of VE user interaction components. Our use of this 
approach with a real-world battlefield visualization VE has re- 
sulted in a VE for which we have empirical evidence of effec- 
tiveness and usability. 
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Abstract 

In this paper we describe a battlefield visualization system, called 
Dragon, which we have implemented on a virtual reality responsive 
workbench. The Dragon system has been successfully deployed as 
part of two large military exercises: the Hunter Warrior advanced 
warfighting experiment, in March 1997, and the Joint Counter Mine 
advanced concept tactical demonstration, in August and September 
1997. We describe battlefield visualization, the Dragon system, and 
the workbench* and we describe our experiences as part of these two 
real-world deployments, with an emphasis on lessons learned and 
needed future work. 

Keywords: Battlefield Visualization, Responsive Workbench, Vir- 
tual Reality, Virtual Environments 



1 Introduction 

When fighting a battle, commanders must analyze and understand 
current and future combat situations in order to make good strategic 
decisions. This problem, which is as old as warfare itself, is referred 
to as command and control. In addition, commanders must plan 
and evaluate possible future strategic force movements, an opera- 
tion referred to as planning and shaping. Currently, both activities 
are accomplished with paper maps of the battle area placed under 
sheets of acetate. Technicians receiving intelligence reports from 
the field depict the changing situation with grease pencils. Com- 
manders may then plan various scenarios by drawing additional 
symbology on the map. 

This is a cumbersome, time consuming process: detailed maps 
and overlays can take several hours to print and distribute. The 
fast-changing modern battlefield frequently produces so much time- 
critical information that the above manual techniques are inade- 
quate for properly visualizing the battlespace. At the Naval Re- 
search Laboratory, we have developed a virtual-reality battlefield 
visualization system, termed Dragon, which is implemented on a 
virtual reality responsive workbench. We have found the work- 
bench to be an effective virtual reality interface for a battlefield 
visualization system. 

In this paper we briefly discuss the battlefield visualization prob- 
lem. We describe the workbench and review relevant work done 
to date. We follow this with a brief discussion of various design is- 
sues and tradeoffs we considered as we developed Dragon. We then 
describe using the system as part of two real- world, large-scale mil- 
itary exercises, and point out many of the lessons learned. 
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2 Battlefield Visualization 

Despite the advent of computers and sophisticated decision-making 
software in combat operation centers, the military still undertakes 
battlefield visualization predominantly with paper maps and acetate 
overlays. This Is a hold-over from the days when reports from the 
battlefield arrived at the combat operations center exclusively by 
voice over a radio network. A radio operator at the center received 
the verbal report, and then translated the information into symbol- 
ogy that was hand-drawn on a paper map. Currently, this same data 
is sometimes entered by hand into a computer system, where it can 
be used by computerized battlefield visualization systems* Obvi- 
ously this is a rime-consuming process, with many opportunities 
for introducing error into the data stream. 

New advances in distributed, encrypted digital data links allow 
combat units to report to the combat operations center using com- 
puter networks in place of voice radio links. The intelligence data 
is now available directly in a digital format. No time or manpower 
is wasted translating the data from voice report to computer input 
Avenues for introducing error are also reduced to just the original 
reporter in the field. However, current combat operations centers 
do not take full advantage of this digital data. Time and manpower 
is spent monitoring this digital data stream and translating it into 
symbology on a paper map. 

3 The Responsive Workbench 

The Naval Research Laboratory's virtual reality responsive work- 
bench [6, 10] provides a three-dimensional display for observ- 
ing and managing battlespace information. The workbench pro- 
vides a natural metaphor for visualizing and interacting with 3D 
computer-generated scenes using a familiar tabletop environment. 
Applications which traditionally require personnel to collaborate 
around a table are excellent candidates for using the workbench. 
Since 1994, the Naval Research Laboratory has successfully devel- 
oped workbench-based prototype systems for medical planning and 
training [8], simulation based design, and battlefield visualization 
for planning arid shaping as well as command and control [1,9], 

4 The Dragon System 

The Dragon battlefield visualization system runs on a virtual reality 
responsive workbench. The system displays a three-dimensional 
representation of the battlespace (see Color Plates), which includes 
a terrain map, entities representing friendly, enemy, unknown, and 
neutral units, and symbology representing other features such as 
obstructions or key points in the plan of operations. Entities are 
represented both by schematic models as well as standard battle- 
field visualization symbols. Dragon receives electronic intelligence 
feeds which relate each entity's current status, including such in- 
formation as position, current speed and heading, current damage 
condition, and so forth. As these reports are received, Dragon up- 
dates the corresponding models on the map. Users can view the 
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battlespace in either monographic or stereographic mode, and navi- 
gate to observe the map and entities from any angle and orientation. 
They can also query and manipulate the entities. 

4.1 Interaction Techniques 

A fundamental design decision for any virtual environment is how 
users navigate through the environment and interact with objects in 
the environment. 

4.1 .1 The Virtual Laser Pointer 

The Naval Research Laboratory has developed three general in- 
teraction methods for the workbench: gesture recognition using a 
pinchglovc [7], speech recognition, and a hand-held joystick. We 
considered using each of these as an input device for the Dragon 
system. Although an interesting avenue for virtual environment in- 
teraction, we deemed the speech recognition technology to be too 
immature for battlefield visualization. We also found the pinch- 
glove problematic — it is fragile, time-consuming to pass from user 
to user, and only works for right-handed users whose hands are ap- 
proximately the same size. In contrast, the hand-held joystick is 
relatively robust and very quickly handed from user to user, and 
works for both right- and left-handed users. 

For the Dragon system we modified a three-button game joystick 
by removing it from its base and placing a six degree-of-freedom 
position sensor inside. The joystick's position and orientation arc 
tracked relative to an emitter located on the front center of the work- 
bench. The interaction metaphor for this joystick is a virtual laser 
pointer. We imagine that a laser beam comes out of the front' of the 
joystick and enters the virtual environment Our system renders this 
beam as another virtual object (see Color Plate 3). Where the beam 
intersects the terrain or objects, a highlighted marker appears. 

4.1.2 Navigation Metaphors Investigated 

Using the virtual laser pointer as an interaction device, we im- 
plemented and field-tested two virtual environment navigation 
metaphors. 

One metaphor, termed map-centric navigation, was based on 
how users interact with a real physical map placed on a table sur- 
face. Various button combinations produce three navigation modes: 
pan, zoom, and pitch/yaw. For each mode, the map mimics the mo- 
tion of the joystick. That is, the map acts as if it were attached to the 
joystick: a motion along a vector by the joystick causes the map to 
move by that same vector. For this metaphor the user makes a zero- 
order control gesture — that is, the magnitude of the user's gesture 
controls the distance of the virtual motion. This means that, for 
example, when panning from one side to the other of a zoomed-in 
map, the user must make repeated panning gestures, each of which 
translates the map. a distance equivalent to the length of the user's 
gesture. 

The other navigation metaphor we investigated, termed user* 
centric navigation, was loosely based on the metaphor of a user 
flying above the map as if in an airplane. Various button combina- 
tions again produce three navigation modes: pan/zoom, pitch/yaw, 
and rotate/zoom. For the user-centric navigation the user makes a 
first-order control gesture — that is, the magnitude of the user's ges- 
ture controls the velocity of the virtual motion. This means that, for 
example, the user can fly from one side to the other of a zoomed-in 
map with a single gesture. 

4.1.3 Object Manipulation 

The user interacts with all entities on the map with the virtual laser 
pointer. The user selects an entity simply by pointing at it. Entity 
selection is denoted by drawing a blue wire frame sphere around 



the entity (see Color Plate 3). When an entity is selected, a window 
pops up on the right side of the workbench with all of the known 
information about that entity gathered from the system. By pressing 
a button on the joystick, the user can pick up a selected entity and 
move it around the virtual environment. 

4.2 Models and Symbology 

We use two different schemes for representing entities on the map 
(sec Color Plates 2 and 3). For some entities we used Intelli- 
gence Preparation of the Battlefield (TPB) symbology [5]. This is a 
military-standard set of symbols representing both force units (e.g. 
companies of troops) as well as particular areas or locations (e.g. 
a named area of interest or a targeted area of interest). Since we 
needed 3D objects that were visible from oblique angles, we ex- 
truded the 2D symbols into cubes, and texture-mapped the symbols 
onto each face (for example, in Color Plate 2 the boxes marked with 
blue and red 'Vs" represent troop squads). For other entities, such 
as tanks, ships, and planes, we used realistic 3D models, both be- 
cause we felt that an operator would be able to rapidly identify a 
realistic model based on shape and coloring, and because the IPB 
standard lacks symbology for specific pieces of hardware. 

When rendered at a real-world size the entities all quickly be- 
came invisible as the user zooms away from the map (see Color 
Plate 1), Therefore, we provided a user-controllable scaling factor 
for all entities. In addition, most of the entities were represented at 
multiple !evels-of-detau\ which increased rendering efficiency. Fi- 
nally, some entities supported multiple model versions representing 
variants on the basic chassis, such as a command variant, as well as 
various levels of damage. 

Entity allegiance was multiply encoded using color, shading, and 
textures. Friendly units were tighter hued or blue in color and con- 
tained at least one American Mag somewhere on the unit. Enemy 
units were darker or red in color and flew a skull and cross bones, 
flag. Although to date the exercises where we have used the Dragon 
. system have not required it, it is necessary to develop an additional 
encoding for unknown, neutral, and civilian units. 

4.3 Data Feeds 

Currently, the US military typically uses the Global Command and 
Control System (GCCS) [2] for collecting, storing, visualizing, and 
interacting with data coming from the field. This data is also occa- 
sionally translated into the Distributed Interactive Simulation (DIS) 
[4] format for use in military simulation systems such as the MOD- 
ular Synthetic Armed Forces (MODSAF) system [1 1J. DIS systems 
are often used to simulate the outcome of a given situation and plan. 
Both systems provide position and status information for each en- 
tity in the battlespace. 

Dragon can receive data feeds in both GCCS and DIS formats. 
Additional information, such as planning symbology, special enu- 
meration of features, and hazards on the battlefield are hand placed 
by the user, either interactively or through a simple text file. 

5 Lessons from the Field 

The Dragon system and the workbench have been successfully de- 
ployed as a prototype system at two different military operations 
during the past year, the Hunter Warrior advanced warfighting ex- 
periment in March 1997 and the Joint Counter Mine advanced con- 
cept tactical demonstration in August and September 1997 [I]. 

The intent of the Hunter Warrior demonstration was to prove the 
potential of using a workbench-based battlefield visualization sys- 
tem to provide situational awareness as well as support for con- 
ducting planning and shaping operations. The workbench was po- 
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sitioned in the planning and shaping section of the combat oper- 
ations center and used continuously to brief VIPs, both civilian 
and military. The commanders were very impressed by the ability 
to visualize the current operating picture accurately and efficiently 
on the workbench, especially when compared to the traditional but 
manpower- and time-intensive technique of using a paper map with 
acetate overlays. 

The intent of the Joint Counter Mine demonstration was to show- 
case the potential of the workbench to another user community 
within the military that was concentrating their efforts on command 
and control of units in a highly congested operation area. For this 
exercise, the workbench displayed the ongoing simulation of new 
tactics and equipment for overcoming enemy mines. 

5.1 Data Feeds 

The GCCS-M system (the Marine variant of GCCS [2]) was used 
during the Hunter Warrior advanced warfighting experiment. Units 
in the field created digital report messages on Apple Newton per- 
sonal data assistants, which in turn were linked back to the combat 
operations center by a radio wide-area network. The messages were 
parsed into a form that could be used by GCCS-M. The Dragon sys- 
tem received update reports from GCCS-M at regular intervals or 
upon user demand. Dragon parsed the GCCS-M data stream for 
unit type, positional data, and the last textual message sent from 
the unit (see Color Plate 3). Since the source of the GCCS-M in- 
formation stream was from units in the field entering data, the data 
feed on individual entities was very irregular and sparse, resulting 
in "jerky" entity movements. 

The DIS protocol [4] was used at the Joint Counter Mine demon- 
stration. Although the data feed per unit was also irregular, be- 
cause DIS contains a built-in protocol for dead reckoning, the entity 
movement was smoother and less distracting than it was at Hunter 
Warrior. This demonstrated that a workbench-based battlefield vi- 
sualization system could also effectively provide situational aware- 
ness for a simul ated military environment 



5.2 Interaction 

We initially thought that a battlespace visualization system only re- 
quired a map-centric navigation metaphor. We based this decision 
on our observations of how users interact with maps in the combat 
operations center. In reality, the Dragon system and workbench cre- 
ate a very rich environment in which users can do much more than 
just move a map. They can actually experience the environment by 
visually sizing up terrain features, entity placement, fields of fire, 
lines of sight, etc. Map-centric navigation worked well when glob- 
ally manipulating the environment and conducting command and 
control operations on large-scale units. However, when small-scale 
operations were being examined, the operators wanted to be able to 
**fly M over the terrain and even get down to a first person viewpoint 
as if they were a person walking around on the ground. Map-centric 
navigation did not lend itself to conducting these types of opera- 
tions in an intuitive way, which encouraged our development of the 
user-centric navigation metaphor. 

The Dragon system lets the user select either a monographic or 
a stereographic view of the batdespace. In these exercises we ob- 
served that users chose the monographic mode much more often 
than the stereographic mode. We believe there are three main rea- 
sons for this; 1) the display technology currently only supports one 
(or at most two) stereo users. 2) stereo is fatiguing to the user, and 
3) the environments for both military exercises were both relatively 
fiat, and thus the improved depth perception from the stereographic 
mode was not as important as it may become in more geographi- 
cally varied environments. 



5.3 Visualizing the Battlespace 

Displaying thousands of entities on a textured terrain map is a dif- 
ficult visualization problem. In particular, it is difficult for a user 
to discriminate between various battlefield entities. Mitigating this 
problem has required experimentation, which has taught us several 
valuable lessons. 

Camouflage: Many of our early 3D entity models had realistic 
camouflage texturing. Friendly units had lighter camouflage pat- 
terns and enemy units had darker camouflage patterns. This was 
often too subtle of a difference for differentiating between friend 
and enemy. In addition, the first terrain texture maps chosen con- 
tained a significant component of green, thus providing an ideal 
background for the camouflaged models to blend into. This, com- 
bined with the difficulty of picking appropriate model sizes, made 
it difficult to locate and identify the models on the terrain surface. 
One solution was to use a gray-scale texture map for the terrain. 
This highlighted the camouflaged models greatly, but reduced the 
ability to display terrain information using color as the discrimina- 
tor. 

Model Variation: There are usually multiple variations on a 
given military hardware entity. For example, an amphibious assault 
vehicle might come in a troop carrier configuration, a command and 
control variation, or an attack configuration complete with a light 
cannon. A battlefield visualization system must be able to visually 
differentiate between the different configurations of each entity. A 
related problem is that the ability to differentiate between entities 
can become difficult if the external appearance between two mod- 
els is too similar. This calls for more modeling than we provided 
with Dragon, potentially with a standard model format that supports 
multiple variations within each model. 

Scaling and Aggregation: There are obvious problems with 
attempting to display every individual entity in the battlespace. One 
solution is to aggregate individual units into larger hierarchical units 
13]. Another solution is to scale the entities up as one zooms out, 
and down as one zooms in. However, scaling makes the aggregation 
problem even worse, as even distant entities will eventually inter- 
sect if one zooms out far enough. Further, the models will occlude 
the terrain beneath, and with a large size, it is difficult to ascertain 
a model's true position. 

These are all very difficult visualization problems for which we do 
not yet have adequate solutions. Symbology, standard or new, may 
help with entity identification, scaling, and aggregation. However, 
it is critical that wc do not transfer negative information. Any dis- 
play metaphors must be thought out in detail and potential problems 
clearly documented and then understood by the user. 

5.4 Impact on Battlefield Visualization 

Visualizing the battlefield with the traditional paper map, acetate 
overlays, and grease pencils has served military commanders well 
for many years. However, the labor- and time-intensive procedures 
required to create and maintain these maps translate into expensive 
and out-of-date information being placed in front of the comman- 
der. In addition, the modern battlefield is producing ever more data 
at an ever-quicker rate. Finally, the modern combat operations cen- 
ter contains too many consoles displaying too much specialized in- 
formation. The result is both information overload and information 
fragmentation. Clearly, traditional methods for battlefield visual- 
ization need to be improved. 

GCCS-M is a partial solution to these problems. This system 
interfaces with some of the battlespace information gathering sys- 
tems, and it does have the ability to display its results graphically. 
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However, the display is two-dimensional and is easily cluttered. We 
found the general opinion to be that the GCCS-M user interface is 
cumbersome and difficult to use. 

The goal of Dragon is to improve battlefield visualization by 
using visualization techniques. Id pursuing this goal we have ex- 
tended battlefield visualization into three dimensions, represented 
entities by both symbolic and realistic three-dimensional models, 
displayed the results on a responsive workbench, and provided an 
intuitive interface for navigation and entity manipulation. We have 
also provided a system that integrates the output from several data 
feeds into a uniform representation presented on a single display 
surface. To date our field experiences suggest that Dragon is a su- 
perior battlefield visualization platform, compared to both the tra- 
ditional map-with-overlay method as well as 2D battlefield visual- 
ization systems such as GCCS-M. 



6 Conclusions and Future Work 

Battlefield Visualization in the support of command and control as 
well as planning and shaping activities is a very difficult problem, 
but one that has potential for a large payoff. From our experience 
with Dragon we have come up with a number of areas for future 
work: 

• It is necessary to conduct a formal task analysis to understand 
what different users in the combat operations center are trying 
to accomplish, how each task is currently accomplished, and 
finally how a visualization system can assist in accomplishing 
the tasks more quickly, with less manpower, and with a greater 
level of accuracy. A careful task analysis should identify key 
defaults that can be used to specify everything from how a 
query result should be displayed to what color scheme should 
be used. 

• It is necessary to conduct user studies to investigate all the 
usability characteristics of the Dragon system, with an eye to- 
wards understanding user preferences and improving the user 
interface. Such a series of user studies is currently underway, 
with an emphasis on navigation techniques. 

• Any battlefield necessarily deals with uncertainty, and it is 
necessary to determine ways to represent and encode the con- 
fidence level that exists for each piece of battlefield data. For 
example, as the last reported position of an entity ages, the 
uncertainty of where the entity is currently located grows. 

• Time must also become a part of a battlefield visualization 
system. This might be used to play back the previous 24 hours 
or to store and review the plans for the upcoming 24 hours. 

• Distributed computing is the direction in which all military 

oyotcma, but especially th* Navy and Marines, ate moving. 

This will include remote collaboration as well as distributing 
the work load across multiple platforms. 

• The system must support more data feeds, including new as 
well as legacy systems. In the combat operations center mere 
are still too many consoles with specialized users doing very 
narrowly defined tasks. Military commanders at the exercises 
we attended made it clear that a system capable of visualizing 
the output from a multitude of legacy systems in a single, con- 
sistent display and interface is desperately needed by today's 
combat forces. 
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Abstract 

The Navy under the Airborne Battle 
Management System (ABMS) program, sponsored 
by ONR Code 31 (Command, Control and Combat 
Systems) and executed by NAVAIR, is addressing 
the shifting focus of naval operations to power 
projection in a littoral environment, as expressed in 
"Forward from the Sea" and "Operational 
Maneuver from the Sea". To effectively project 
naval power into the littoral environment with 
minimal reliance on a heavy footprint ashore, the 
force will require extensive NCW support from 
C4ISR located on the "high ground" provided by 
naval aviation. ABMS specifically addresses the 
problem of "How to get the right information / 
image(s) to the right pilot(s) / platform(s) soon 
enough?" in order to provide Navy and Marine 
Corps aviation the ability to observe, direct, and 
control from high above the littoral battlespace. 
This will be central to our ability to respond quickly 
and accurately to threats deep inland and to employ 
ship-launched, air-launched, and ground-launched 
weapons with maximum effect. 

ABMS provides the potential for substantial 
reduction in the engagement timeline for time 
critical and/or mobile targets through the 
development of and/or the integration of critical 
sensor-to-shooter and sensor-to-weapons 
technologies. Specifically, ABMS addresses 
Continuous Surveillance/Battle Damage 
Assessment, Sensor Management, Information 
Gathering & Management, Information 
Dissemination, Target/Combat ID, Multi-mission 
Coordination and Deconfliction, Dynamic ROE / 
ATO and other guidance, Targeting, Target/ 
Weapon Pairing Rate, Flight Path Routing and 
Threat Avoidance, Precision Fire, Automated 
Distributed / Decentralized Weaponeering (C2), and 
Platform Survivability. 

ABMS specifically addresses the problem of 
"How to get the right information / image(s) to the 
right pilot(s) / platform(s) soon enough?" by 



shortening the "Decide" phase of both the Detect, 
Decide, Engage and BDI/BDA Assessment aspects 
of the kill chain and the OODA (Observe, Orient, 
Decide, Act) loop. ABMS is three fold in that it is 
(1) developing and demonstrating the 
implementation concepts for intelligent in-flight 4- 
D (space - time) image management and 
dissemination, namely the "Image Editor" 
(Surveying/Anomaly Detection, Chicleting, IDing, 
& Filtering), "Router" (Selecting, Geo-Sorting, & 
Disseminating), and "Sorter" (Time- 
Sorting/Posting, Geo-Registering to 3-D Terrain 
Database, and Displaying when Relevant) for 
bandwidth / timeline reduction, (2) developing and 
demonstrating Automatic Target Cueing (ATC) and 
Combat Identification (CJD) to recognize 3-D 
objects (discriminating between targets and 
friendlies / neutrals) and discriminating between 3- 
D objects and 2-D decoys, and (3) developing and 
demonstrating the *'plug & play" system integration 
of a tactical airborne Network Centric information / 
image battle management and C2 architecture 
(which includes dynamic ROE, ATO, etc.), geo- 
registration software tools, various image 
processing software applications and ABMS' Image 
Editor/Router/Sorter. 

Background 

Naval aviation represents a unique resource 
beyond the evident use for sensing and strike. This 
is an element of the forward force with minimal 
distance to the enemy, minimal response time to 
threats (both offensively and defensively); location 
on a "virtual high ground" that can be as high as 
Mount Everest, and potential proximity to the 
enemy that is matched only by ground forces in 
close combat. Operational commanders have strived 
for centuries to capture the "high ground" as a 
location from which to observe, command, and 
control the battle, and naval aviation offers a 
significant potential to establish this advantage for 
our forces. 
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These advantages are offset by obvious 
limitations in the ability to support warfighting 
functions in aircraft that can operate from bases 
afloat. Limitations include factors such as saturated 
workload for the pilots and small airborne staffs; 
limited bandwidth, processing, memory, and 
display resolution/definition; increased tempo/rate 
of change of the battlespace when conflict breaks 
out; and determining/providing functionally tailored 
information to the warfighters (what the warfighter 
needs) when they need it. Other issues include 
obtaining and integrating off-board and on-board 
information/images (12), the need for immediately 
comprehendible information, "time-late" 
information, skin-to-skin architectural implications, 
etc., real-time (RT) node characteristics within the 
network-centric (NC) architecture for an embedded 
information management system (IMS) in the 
platform to manage sharing information (e.g., 
graphics & imagery, air/ground threats, 
retasking/replanning) with other forces to support a 
RT all-source 12 based sensor-to-C2-to-shooter 
picture of the battlespace, and affordability and 
retrofit issues (including shortfalls and architectural 
limitations of legacy aircraft). There is also the need 
to incorporate the architecture and infrastructure to 
produce an enterprise wide low latency tactical 
Single Integrated Battlespace Picture (SIBP). 
Another major issue is the absence of an integrated 
"plug & play" tactical airborne Network Centric 12 
battle management and C2 architecture for the 
above critical sensor-to-shooter and sensor-to- 
weapons technologies. 

Discussion 

ABMS addresses these issues by a software 
development and implementation approach that is 
coordinated with preplanned software upgrades in 
the aircraft. One important software enhancement 
will address the current difficulty that Naval air 
assets have in sharing images that they collect 
during their respective missions. This is an 
important shortfall for time critical strike since 
tactical aircraft have the potential to share the most 
tactically relevant and lowest latency 12 and 
substantially reduce the timeline associated with the 
detection, decision, engagement and assessment of 
battle damage. An important technology area that 
the Airborne Battle Management System addresses 
is information management and dissemination, 



namely the processing and management to 
automatically handle the vast amount of data 
collected, and to automatically filter and 
disseminate the right information to the right 
operators at the right time. This includes merging 
of multimedia information as well as fragmenting 
massive files such as imagery to deliver the part(s) 
of the images that are relevant to the task. 

One is forced to ask (since we can broadcast 
voice transmissions today): "Why can't we solve 
the problem of sending the right image to the right 
platform today?" A brute force approach does not 
work due to bandwidth overload. Existing 
demonstrations have essentially been limited to 
preplanned point-to-point transmissions. 

A number of ongoing programs can be brought 
to bear immediately to provide near term 
enhancements. 

The Advanced Aviation Subsystems (AAS) 
program, sponsored by ONR Code 35, has 
developed 3-D visualization technology making 
direct use of geo-specific databases and image base 
data. The principle functional product from AAS 
was used in Kosovo to provide targeting 
information from UAV (unmanned air vehicle) 
video in near real time. It provides the foundation 
for image exploitation and the Common Tactical 
Picture (CTP) by accurately geo-registering 
imagery and. presenting it in a wider contextual 
view. In FY'OO and '01, the visualization and image 
registration capability will be implemented in 
embedded avionics processing hardware and then 
demonstrate these technologies in a time critical 
strike scenario. 

The Real Time Execution Decision Support 
(REDS) Program was initiated by ONR to provide 
near real time retargeting capability and to expedite 
the processing and use of air tasking and air control 
plans. REDS has produced initial capabilities that 
are being used and evaluated onboard aircraft 
carriers and will transition to the future Joint 
Mission Planning System. REDS also processes the 
ATO and can deliver parsed, formatted data to the 
tactical control systems in aircraft such as the E-2C 
REDS offers further potential to develop and 
upload planned route data in minute by minute 
snapshots for near real time deconfliction and 
airspace management. 
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The Automated Rules of Engagement (AROE) 
program initiated by ONR in FY99 will provide 
capability to automatically parse and process ROE 
messages. This will include a dynamic ROE an 
ATO database that will be linked to the tactical 
entities in the situation databases and will allow the 
operators to have a consistent and up-to-date view 
of the ATO and ROE implications on the threats. 
AROE offers an opportunity to offload a significant 
workload from the operators to this automation. 
Automated agents can provide further assistance by 
tracking ATO and ROE and providing cues and 
alerts to the tactical commanders and controllers 
and also to the higher level commands that are 
responsible for establishing the ATO and ROE. 

Tactical Control System (TCS) is being 
developed to provide a wide range of control 
capabilities for UAVs. An airborne C4ISR control 
post is clearly a high value location to control low 
altitude UAVs deep in enemy territory since the 
links can be direct and relatively wideband and low 
latency. Proposed enhancements to this system to 
support time critical strikes will use advanced 
software to decrease the required footprint and 
operator-intensive workload and to make the system 
more amenable to operation onboard tactical 
aircraft that can operate from afloat bases. This 
could include both large fixed wing aircraft and 
VTOL aircraft such as the MV-22 or helicopters. 

LARIAT 2, started in FTOO is integrating the 
AAS visualization toolset into NAWCAD's Flying 
testbed, the Hairy Buffalo, and integrating on-board 
sensors and communication systems with AAS. 
Since AAS and the DARPA sponsored Warfighter 
Visualization program focussed on the EO and IR 
spectrum, LARIAT 2 will principally focus on the 
accurate registration of SAR imagery. Integration of 
SAR with highly accurate geo-specific data will 
provide an all weather strike capability. 

The ONR ABMS program was initiated in 
FY99 to develop/demonstrate an affordable NC 
IMS and battle management / C2 architecture that 
supports the unique Naval Aviation C4ISR&T 
requirements for mission planning/replanning and 
to process/deliver/display RT tactically relevant 12 
for RT targeting, air strike, and BDA/L The 
integrated technology offers major improvements in 
time critical targeting, NC IMS, battle management 
and C2, aircraft survivability, and lethality through 



the use of advanced information technology to 
enhance the ability to differentiate targets using 
existing sensors, to support engagement/weapons 
release, to shorten the OODA loop, to improve real 
time situation awareness to avoid pop-up and air 
threats, to support rapid replanning of strike 
packages for time critical targets, to decrease 
fratricide and unintended collateral damage, to 
deconflict missions in the air and in the objective 
area on the ground, and to enhance the ability to 
dominate the battlespace. 

ABMS supports the goals and objectives of the 
Navy Integrated Warfare Architecture (IWAR) by 
supporting several future Naval enabling 
capabilities and transition technologies in the areas 
of information superiority, power projection and air 
dominance. Furthermore, ABMS directly supports 
the ONR mission of developing supporting 
technologies and science and technology programs 
to provide such future Naval enabling capabilities. 
ABMS directly supports ONR's recent initiatives in 
Time Critical Strike while also supporting other 
ONR initiatives, e.g., Decision Support Systems, 
Autonomous Operations, Network Centric Warfare, . 
Information Distribution, Missile Defense, and 
Platform Protection. In addition, ABMS supports 
ONR's mission of supporting several of the 
emphasis areas identified by Naval Aviation, e.g., 
Combat ID, ISR & Air-to-Ground Targeting, 
Network Connectivity, Tactical situation awareness 
(SA), EW/Defensive CM, Air-to-Ground Weapons, 
and Total SA. 

Approach 

To resolve the various issues identified above, 
ABMS is taking a three-fold approach: (1) 
development, application, and/or implementation of 
concepts for intelligent 4-D (space - time) in-flight 
multimedia information management for substantial 
bandwidth / timeline reduction and improved 
decision support, including where appropriate the 
integration of the ground picture with the air picture 
to create one SIBP; (2) system engineering analysis 
for implementation and integration of the ABMS 
concept into appropriate aircraft (E-2C, F/A-18, 
UAVs, MV-22, Marine Corps helicopters, etc.) to 
provide tactical airborne Network Centric battle 
management command and control capability 
(including determining what software elements are 
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needed / desirable in each platform) and (3) the 
Phase 2, laboratory demonstrations using 
simulations and flight demonstrations, to 
demonstrate the tactical utility of the ABMS 
concept. ABMS reduces the engagement timeline to 
minutes by the design, development and integration 
of its own technology with other existing and/or 
currently being developed technologies (i.e., AAS, 
REDS, AROE, TCS, LARIAT 2 and industrial 
efforts). Our approach is to develop the 
implementation concepts for intelligent in-flight 4- 
D image management and dissemination through 
software and perform the system integration of a 
•'plug & play" tactical airborne Network Centric 12 
battle management and C2 architecture for time 
critical sensor-to-shooter and sensor-to- weapons 
technologies. We will then demonstrate the ABMS 
architecture that enables shared airborne and 
surface platform information to Detect, Identify, 
Decide, Engage/Kill time critical fixed, mobile and 
moving targets while providing BDI/BDA in an all 
weather environment using multiple sensors and 
multiple existing (and/or new) communications 
links. The demonstrations will also show the use of 
that shared information for full S A for air/ground 
threat avoidance and for replanning consistent with 
the ATO and ROE. The transition products are (1) 
basically software upgrades (no new links, no new 
boxes, but will involve some new wiring) to 
provide the intelligent in-flight 4-D image 
management and dissemination, (2) an integrated 
"plug & play" tactical airborne Network Centric 
information / image battle management and C2 
architecture, (3) airborne planning/replanning 
system, (4) theater wide cueing of imaging sensors, 
(5) automatic decision aids including dynamic ROE 
and ATO, (6) analysis of ATC/ATR alternatives, 
(7) analysis and dissemination of offboard land 
surveillance sensors (UGS), and (8) distributed 
weapons coordination and collaborative 
weaponeering of F/A-18s. The intelligent in-flight 
4-D image management and dissemination portion 
of ABMS is a 6.2 research project and thus involves 
risk. Hence, in FY99 the "seed" ABMS program 
was established to mitigate this risk. The proof-of- 
concept 4-D intelligent image management and 
dissemination algorithm laboratory demonstration 
in early FYOO was successful in mitigating this 
portion of the risk. In addition, the ATC / CID 
laboratory demonstration in mid-FYOO was 
successful in mitigating this portion of the risk 



Likewise, both the intelligent in-flight 4-D image 
management and dissemination portion of ABMS 
and the ATC / CID as well as the integrated "plug 
& play" tactical airborne Network Centric 
information / image battle management and C2 
architecture leverage other on-going ONR activities 
to further mitigate risk. 

Description 

ABMS is three fold in that it is (1) developing 
and demonstrating the implementation concepts for 
intelligent in-flight 4-D (space - time) image 
management and dissemination, namely the "Image 
Editor" (SurveyingTHot Spot" Detection, 
Chicleting, IDing, & Filtering), "Router" 
(Selecting, Geo-Sorting, & Disseminating), and 
"Sorter" (Time-Sorting/Posting, Geo-Registering to 
3-D Terrain Database, and Displaying when 
Relevant) for bandwidth / timeline reduction 
(Figure 1), (2) developing and demonstrating ATC 
and CED to recognize 3-D objects (discriminating 
between targets and friendlies / neutrals) and 
discriminating between 3-D objects and 2-D decoys 
(Figure 2 depicts the ABMS Concept), and (3) 
developing and demonstrating the "plug & play" 
system integration of an architecture for tactical 
airborne Network Centric information / image battle 
management and C2 (which includes dynamic 
ROE, ATO, etc.), geo-registration software tools, 
various image processing software applications and 
ABMS' Image Editor/Router/Sorter. The ABMS 
Functional Architecture is depicted in Figure 3 and 
the Program is depicted below in Figure 4. 

ABMS substantial reduces the engagement 
timeline for time critical and/or mobile targets 
through the development of and/or the integration 
of critical sensor-to-shooter and sensor-to- weapons 
technologies. Specifically, ABMS addresses: 
Continuous Surveillance/Battle Damage 
Assessment, Information Gathering & 
Management, Information Dissemination, 
Deconfliction, Target/Combat ID, Targeting, 
Target/Weapon Pairing Rate, Precision Fire, 
Automated Distributed/ Decentralized 
Weaponeering (C2), and Platform Survivability. 

Time Critical Strike (TCS) requires the 
capability to quickly transfer critical information 
among the various military forces. Even more 
important, however, is that the information which is 
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passed be relevant to the mission and that this 
relevance be made abundantly clear to the 
warfighter. Therefore, our methodology is to handle 
both the flow of critical information with its 
relevance and integrate this into the "plug & play" 
tactical airborne Network Centric information / 
image battle management and C2 system . Figure 5 
depicts the ABMS low latency 12 flow and real- 
time execution control. 

Through the "Image Editor", "Router" and 
"Sorter" of the intelligent in-flight 4-D image 
management and dissemination subsystem, the 
ATC / CID software and the in-flight, in-theater 
integrated "plug & play" tactical airborne Network 
Centric information / image battle management and 
C2 system the time critical targeting timelines are 
reduced and targeting accuracy is increased. 
Information will automatically be routed through 
the intelligent network and directed to those 
warfighters with an operational need for it. The 
information will then be presented in a manner 
tailored to the warfighter' s needs. Since only a 
small portion of the information is likely to be 
relevant to individual warfighting components, the 
actual information / image flow necessary to 
accomplish this is much smaller than might be 
imagined. The quantitative benefit is a 100X to 
10.000X improvement in the bandwidth / timeline 
to pass the image chiclets to the image analysts and 
/or warfighters. ABMS is an affordable solution 
in that it utilizes existing Data Links to provide an 
integrated tactical airborne Network Centric 
information / image battle management and C2 
system which can provide one meter targeting 
accuracy through the use of a 3-D Digital Terrain 
Elevation Database. Figure 6 depicts our global 
vision of the Network Centric C4ISR&T 
Information Superiority Architecture and the CTP / 
COP-SD3P. 

To solve these problems we were forced into a 
new paradigm, i.e. a "smart" approach that 
necessitates ( 1) innovative, intelligent RT 4-D 
(space-time) image management and dissemination 
utilizing logic, down-selection, determination of 
applicable user(s) and intelligent Image Editing, 
Routing, and Sorting. ABMS reduces the 
engagement timeline to minutes by this new 
paradigm and leveraging of other existing and/or 
current developing technologies. Our approach is to 



develop the implementation concepts for intelligent 
in-flight 4-D image management and dissemination 
and ATC / CID through software. ABMS performs 
the system integration of a "plug & play" tactical 
airborne Network Centric information / image battle 
management and C2 architecture for time critical 
sensor-to-shooter and sensor-to-weapons 
technologies. 

Payoff 

The ABMS expected payoffs are: (a) Timely 
processing of images (e.g. selects, filters, shares, 
routes, delivers, geo-registers) and display of low 
latency tactically relevant images for real-time 
targeting, air strike, BDI / BD A and (total and 
tactical) S A for a Common Tactical / Operational 
Picture (CTP / COP) and time critical targeting; (b) 
ATC / CID, (c) Handles both the flow of critical 
information and its relevance as well as managing 
the available limited network bandwidth (BW); (d) 
Shrinks the OODA cycle time by moving tactical 
functions from the rear-base to airborne platforms; 
(e) Provides GPS equivalent targeting through geo- 
registration of high quality image chiclets into a 
Digital Terrain Elevation Data (DTED) type 3-D 
data base; (f) Supports mission planning / 
replanning; (g) Anticipates threats (warning the 
pilots and decluttering cockpit displays), and (h) 
Provides an integrated "plug & play" tactical 
airborne Network Centric information / image battle 
management and C2 system for time critical 
targeting. In short, ABMS will ensure that the right 
information gets delivered to the right air platform 
at the right time and is presented / displayed when 
relevant 

Summary 

ABMS will provide tactical flexibility through 
the implementation of technologies that will 
accelerate the transfer of useful information to the 
tactical warfighter and will integrate leveraging 
technologies to address existing shortfalls to 
improve the warfighters ability to prosecute re- 
locatable non-emitting targets (TBM, TEL, CCM), 
short dwell mobile intermittently emitting targets 
(TAC, SAM, IADS), and moving targets (tanks, 
APV, trucks). ABMS will provide the ability to 
discriminate between tanks, trucks, friendlies, 
neutrals and 2-D decoys. With 80% of the target set 
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mobile / re-locatable, ABMS will improve the 
Warfighter's reaction time by dynamically 
assessing the changed battlespace. ABMS will 
provide an affordable information manage and 
disseminate system that selects, processes and 



delivers the right information / images to the right 
platform at the right time and displays / presents it 
when relevant, shrinks the OODA loop cycle time, 
and provides a CTP / COP - a SIBP. 



FIGURE 1: ABMS IMPLEMENTATION / CONCEPT: 
INTELLIGENT IN-FLIGHT 4-D (SPACE - TIME) IMAGE 
MANAGEMENT 
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Figure 2 ABMS Concept / Example 
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Figure 3 
ABMS' Functional 
Architecture 
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Figure 5: ABMS LOW LATENCY INFORMATION/IMAGE 
FLOW AND REAL-TIME EXECUTION CONTROL 
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